System and method for enhancing the binaural representation for hearing-impaired subjects

ABSTRACT

A method of enhancing binaural representation for a subject includes receiving a first signal and a second signal in response to a plurality of sound sources, generating a number of estimated interaural time differences using the first signal and the second signal, converting each of the number of estimated interaural time differences to a corresponding interaural level difference, using one or more of the corresponding interaural level differences to generate an adjusted first signal, and using the adjusted first signal to generate a number of signals delivered to the subject for enhancing the hearing of the subject.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority under 35 U.S.C. §119(e) from U.S.provisional patent application no. 61/760,401, entitled “Method toEnhance the Binaural Representation for Hearing-Impaired Users” andfiled on Feb. 4, 2013, the contents of which are incorporated herein byreference.

GOVERNMENT CONTRACT

This invention was made with government support under grant #DC008329awarded by the National Institutes of Health. The government has certainrights in the invention.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention pertains to systems and methods for enhancing thehearing of hearing-impaired subjects, and in particular, in theexemplary embodiment to a system and method for enhancing the binauralrepresentation for hearing-impaired subjects.

2. Description of the Related Art

There are two main binaural cues for localizing sound in the plane ofazimuth, namely (i) interaural time differences (ITDs), and (ii)interaural level differences (IUDs). ILDs are a high-frequency cue, andoccur because a sound that is off to one side of a listener's head islouder at the near ear than it is at the far ear (see FIG. 1 discussedherein). ITDs urea low-frequency cue, and are comprised of very smalldifferences in the time-of-arrival between the two ears when a sound isoff to one side.

An auditory environment containing one or more sound sources (talkers orother sources of sound) is often referred to as an auditory scene.Individuals with normal hearing can use the binaural cues discussedabove for sound localization to improve speech intelligibility in anauditory scene where one talker (referred to as a target) and a secondtalker (referred to as a masker) are spatially separated from oneanother, such as in the well-known “cocktail-party” problem. Such animprovement in speech intelligibility is referred to in the art as aspatial release from masking (SRM).

Although listeners with hearing impairment can often understand speechin quiet settings as well as listeners with normal hearing, they oftenshow dramatic declines in speech understanding in the presence ofbackground noise, A competing talker is often the most challenging typeof background noise, and is also the most difficult to ameliorate withtypical noise-reduction schemes, More specifically, individuals withhearing impairment show little SRM. As a result, these listeners areless able to make sense of an auditory scene that contains multiplesound sources, and are thus typically unable to understand speech in thepresence of noise, such as another talker.

A cochlear implant (CI) is a surgically implanted electronic device thatprovides a sense of sound to a person having a hearing impairment. Insome people, cochlear implants can enable sufficient hearing for betterunderstanding of speech. The quality of sound is different from naturalhearing, with less sound information being received and processed by thebrain. However, many patients are able to hear and understand speech andenvironmental sounds.

Implanting both cochleas of hearing-impaired listeners with cochlearimplants (referred to as bilateral cochlear implants (BCIs)) has becomemore common in recent years. However, implantation is invasive, costly,and can potentially destroy any residual hearing in the ear to beimplanted. A loss of residual hearing can be detrimental to speechreception, even if the amount of hearing is extremely limited. Thus, itis crucial that clear benefits of BCIs over a single device, with orwithout the addition of residual hearing, be established to justify thedecision to implant the second ear. One often-cited potential outcome ofBCI is the ability of such users to perceive and use binaural cues.However, BCI users have thus far shown relatively poor localizationabilities and limited SRM. This is likely because BCI users receivelimited access to binaural cues. First, they perceive only ILDs and notITDs. Second, as shown in FIG. 1, robust ILDs are generally restrictedto frequencies above about 1500-2000 Hz because the longer wavelengthsat lower frequencies are not shadowed by the head. Thus, the binauralrepresentation of BCI users is inconsistent across frequency, and it hasbeen shown that sensitivity. to binaural cues declines with such aninconsistency. In addition, any ILDs that BCI users receive will besubjected to large amounts of compression in the processing electronicsof the BCIs. This includes automatic gain control on the processingfrontend, which essentially limits the level of more intense sounds,likely reducing ILDs as a result, and the compression that occurs to mapthe input dynamic range (which is typically 60 dB or less) to theelectric dynamic range (typically 10-20 dB).

There is thus a need for an effective system and method for enhancingthe binaural representation for hearing-impaired subjects, such as,without limitation, those subject that have BCIs.

SUMMARY OF THE INVENTION

In one embodiment, a method of enhancing binaural representation for asubject is provided that includes receiving a first signal and a secondsignal in response to a plurality of sound sources, generating a numberof estimated interaural time differences using the first signal and thesecond signal, converting each of the number of estimated interauraltime differences to a corresponding interaural level difference, usingone or more of the corresponding interaural level differences togenerate an adjusted first signal, and using the adjusted first signalto generate a number of signals delivered to the subject for enhancingthe hearing of the subject.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a graph plotting ILDs versus frequency in the prior art;

FIG. 2 is a schematic diagram of a system for enhancing the binauralrepresentation for hearing-impaired subjects according to one exemplary,non-limiting embodiment of the disclosed concept;

FIG. 3 is a flowchart of a method for enhancing the binauralrepresentation for hearing-impaired subjects according to one exemplary,non-limiting embodiment of the disclosed concept that may be implementedin the system of FIG. 2.

DETAILED DESCRIPTION OF EXEMPLARY EMBODIMENTS

As used herein, the singular form of “a”, “an”, and “the” include pluralreferences unless the context clearly dictates otherwise.

As used herein, the statement that two or more parts or elements are“coupled” shall mean that the parts are joined or operate togethereither directly or indirectly, i.e., through one or more intermediateparts or elements, so long as a link occurs.

As used herein, “directly coupled” means that two elements are directlyin contact with each other.

As used herein, “fixedly coupled” or “fixed” means that two elements arecoupled so as to move as one while maintaining a constant orientationrelative to each other.

As used herein, the word “unitary” means a part is created as a singlepiece or unit. That is, a part that includes pieces that are createdseparately and then coupled together as a unit is not a “unitary” partor body.

As employed herein, the statement that two or more parts or elements“engage” one another shall mean that the parts exert a force against oneanother either directly or through one or more intermediate parts orelements.

As employed herein, the tern “number” shall mean one or all integergreater than one (i.e., a plurality).

As used herein, the terms “component” and “system” are intended to referto a computer related entity, either hardware, a combination of hardwareand software, software, or software in execution. For example, acomponent can be, but is not limited to being, a process running on aprocessor, a processor, an object, an executable, a routine, a thread ofexecution, a program, and/or a computer.

As used herein, the term “hearing enhancement device” shall refer to anyelectronic or electroacoustic device structured to enhanced the hearingof a user, and shall include, without limitation, cochlear implants andhearing aids.

Directional phrases used herein, such as, for example and withoutlimitation, top, bottom, left, right, upper, lower, front, back, andderivatives thereof, relate to the orientation of the elements shown inthe drawings and are not limiting upon the claims unless expresslyrecited therein.

FIG. 2 is a schematic diagram of a system 2 for enhancing the binauralrepresentation for hearing-impaired subjects according to one exemplary,non limiting embodiment of the disclosed concept. System 2 is a BCIsystem of a subject that includes a first cochlear implant 4A associatedwith the first (e.g., left) ear of the subject and a second cochlearimplant 4B associated with the second (e.g., right) ear of the subject.As described in greater detail herein, system 2 of the present exemplaryembodiment is structured to enhance the binaural representation for thesubject in order to improve the signal to noise ratio (SNR) for thesubject by estimating ITDs and converting those ITDs to ILDs.

As seen in FIG. 2, cochlear implant 4A includes one or more microphones6A which are structured to pick up sound (such as from one or moretalkers) from the environment surrounding system 2 and generate anelectronic signal representing the sound. Microphone(s) 6A is coupled toa processing module 8A. Processing module 8A comprises a processor 7Aand a memory 9A. Processor 7A may be, for example and withoutlimitation, a microprocessor (μP), a microcontroller, or some othersuitable processing device, that interfaces with memory 9A. Memory 9Acan be any of one or more of a variety of types of internal and/orexternal storage media such as, without limitation, RAM, ROM, EPROM(s),EEPROM(s), FLASH, and the like that provide a storage register, i.e., amachine readable medium, for data storage such as in the fashion of aninternal storage area of a computer, and can be volatile memory ornonvolatile memory. Memory 9A has stored therein a number of routinesthat are executable by processor 7A for processing the signals generatedby microphone(s) 6A in order to produce the signals that are needed tocommunicate sound to the subject. One or more of the routines implement(by way of computer/processor executable instructions) at least oneembodiment of the method described in greater detail below that isconfigured to enhance the binaural representation for hearing-impairedsubjects.

Cochlear implant 4A further includes a transmitter 10A which is coupledto and receives the output from processing module 8A. In the exemplaryembodiment, transmitter 10A is a coil held in position by a magnetplaced behind the external ear, and transmits power and the processedsound signals across the skin to the internal components of cochlearimplant 4A described below by electromagnetic induction.

Cochlear implant 4A still further includes a receiver and stimulatormodule 12A which is secured in bone beneath the skin of the subject.Receiver and stimulator module 12A converts the signals received fromtransmitter 10A into electric impulses and sends them through aninternal cable to electrode array 14A. Electrode array 14A comprises anumber of electrodes wound through the cochlea in the scala tympaniwhich send the impulses to the nerves and then directly to the brainthrough the auditory nerve system.

Finally, cochlear implant 4A includes a short range wirelesscommunications module 16A that is structured and configured to enablecochlear implant 4A to communicate with cochlear implant 4B over a shortrange wireless network.

As seen in FIG. 2, cochlear implant 49 is similar in structure tocochlear implant 4A and includes microphone(s) 6B, processing module 8B(having processor 7B and memory 9B), transmitter 10B, receiver andstimulator module 12B, electrode array 14B, and short range wirelesscommunications module 16A. These components are similar in structure andfunction to the corresponding components of cochlear implant 4Adescribed above. Short range wireless communications module 16B isstructured and configured to enable cochlear implant 4B to communicatewith cochlear implant 4A over a short range wireless network. Thesignificance of this two way communications capability between cochlearimplant 4A and cochlear implant 49 is described elsewhere herein.

FIG. 3 is a flowchart of a method for enhancing the binauralrepresentation for hearing-impaired subjects according to one exemplary,non-limiting embodiment of the disclosed concept. In the exemplaryembodiment described below, the method is described as being implementedin the system of FIG. 2. It will be understood, however, that that ismeant to be exemplary only, and that the method may also be implementedin alternative systems designed to assist the hearing impaired, such assystems employing two hearing aids or even, as discussed elsewhereherein, a system employing a single or no hearing enhancement device. Inaddition, the method of FIG. 3 is designed to be repeated periodicallyby system 2. in the exemplary embodiment, the method is repeated bysystem 2 every 20 μs (although it will be appreciated that otherrepeating time periods are also possible).

Referring to FIG. 3, the system begins at step 20, wherein, in responseto a number of sound sources in an auditory scene (such as, for exampleand without limitation, an auditory scene wherein a target is speakingon one side of the subject and a masker is speaking on the other side ofthe subject), microphone(s) 6A generates and passes to processing module8A a first signal (referred to herein as the “channel A signal”) andmicrophone(s) 6B generates and passes to processing module 8B a secondsignal (referred to herein as the “channel B signal”). Next, at step 22,processing module 8A digitizes the channel A signal and filters it intoa number of frequency bands and processing module 8B digitizes thechannel B signal and filters it into a number of frequency bands. In thenon-limiting exemplary embodiment, each signal is filtered into 32ERB-wide frequency bands, although it will be appreciated that differentnumbers and/or sizes of bands may also be used. In the exemplaryembodiment, processing module 8B then causes the channel B frequencyband data to be transmitted wirelessly to processing module 8A ofcochlear implant 4A using wireless communications module 16B (whichcommunicates with wireless communications module 16A). As a result,processing module 8A will have both the channel A frequency band dataand the channel B frequency band data. Thus, in this example, processingmodule 8A acts as the master and processing module 8B acts as the slave.As will be appreciated, these roles may be reversed, with processingmodule 89 acting as the master and processing module 8A acting as theslave. In another alternative, the channel B signal in its unfilteredform may be wirelessly transmitted to processing module 8A which thenfilters it into the number of frequency bands for further processing asdescribed above.

Next, at step 24, processing module 8A estimates the instantaneous ITDfor (i.e., between) channel A and channel B for each of a number of thefrequency bands. In the exemplary embodiment, processing module 8Aemploys a windowed sliding cross correlation algorithm to estimate eachof the ITDs, In one particular, non-limiting embodiment, the algorithmused a 1.2 ms (±600 μs) window size, and a 22.68 μs step size (1 sampleat 44100 kHz). In each window, the between-channel delay that producedthe largest correlation is taken as the instantaneous ITD. It will beappreciated, however, that the above is meant to be exemplary only andthat alternative techniques/algorithms, such as alternative correlationtechniques/algorithms, may be used to estimate the instantaneous ITD foreach of the frequency bands. Furthermore, in the exemplary embodiment,the processing module 8A estimates the instantaneous ITD for each of thefirst 15 of the 32 frequency bands (i.e., frequencies less then or equalto approximately 1500 Hz). Again, this is not meant to be limiting, asother ones of the frequency bands (e.g., the first 20 or all 32) mayalso be used.

Then, at step 26, each of the estimated ITDs is converted into an ILDbased on a predetermined scheme. For example, the scheme may be aformula or a look-up table that converts each YID value into acorresponding ILD value. In the exemplary embodiment, ±600-μs ITDs areconverted to IUDs of ±30 dB, 0-μs 1TDs are converted to 0-dB ILDs, andintermediate values are linearly interpolated.

Next, at step 28, each of the determined ILDs is used to adjust thevalue of the data of the corresponding frequency band of either thechannel A frequency band data or the channel B frequency band data. Inparticular, in the exemplary embodiment, the determined ILDs are appliedto the channel A and channel B signals by attenuating the signal (i.e.,attenuating the corresponding frequency band data) to the ear (eitherchannel A or channel B) that is contralateral to the apparent locationof the sound as estimated by the corresponding ITD. In the exemplaryembodiment, positive ITDs and ILDs are applied such that the leftchannel is attenuated, and negative ITDs and ILDs are applied such thatthe right channel is attenuated. Thus, for example, if the ITD for agiven frequency band at a given moment in time is estimated to be −300μs (i.e., to the left) (e.g., to channel A), the frequency band for theright ear (e.g., channel B) will be attenuated, for example by 15 dBusing the linear interpolation method. described above.

There are a number of ways in which step 28 may be accomplished insystem 2, For example, once each of the ITDs and ILDs have beendetermined, processing module 8A may perform the required adjustments tothe channel A frequency band data and/or the channel B frequency banddata as needed such that adjusted channel A frequency band data andadjusted channel B frequency band data will then exist. The adjustedchannel frequency band data may then be sent wirelessly to processingmodule 8B for used thereby as described below. Alternatively, processingmodule 8A may generate and wirelessly transmit instructions toprocessing module 8B which indicate to processing module 8B whatadjustments need to be made to create the adjusted channel B frequencyband data in processing module 8B (the adjusted channel A frequency banddata will be generated as needed in processing module 8A). Regardless ofhow this step is done, following step 28, processing module 8A will havethe adjusted channel A frequency band data and processing module 8B willhave the adjusted channel B frequency band data.

Next, at step 30, processing module 8A sums the adjusted channel Afrequency band data to create a channel A output signal and processingmodule 8B sums the adjusted channel B frequency band data to create achannel output signal. Those output signals are then provided totransmitter 8A and transmitter 8B, respectively, and operation continuesfrom there as described elsewhere herein to cause appropriatestimulation signals to be delivered to the subject.

The method just described requires no a priori knowledge of the soundsources. A significant limitation to typical noise reduction approachesis that, in the case of two concurrent talkers, the algorithm has no wayof knowing which is the target and which is the noise, and only onetalker can receive the benefit, to the detriment of the other. The useris not afforded the ability to switch attention from one to the other,as occurs naturally for normal-hearing listeners in naturalmultiple-talker environments. The described method does not suffer fromthis limitation, When two talkers are on either side of the mid-saggitalplane of the listener, both talkers receive a benefit in SNR from themethod presently disclosed, at the near ear. This leaves the listenerfree to attend to one or the other talker, and even switch attention ashe or she likes.

The described method is shown to provide significant benefit to speechunderstanding in the presence of a competing talker, but it should bebeneficial in other types of backgrounds as well, such as steady-statenoise It should also be beneficial with multiple (>2) sound sources.

For hearing impaired listeners who use a single unilateral hearingenhancement device, the potential benefit from the described method maybe enough to warrant the use of a second device that would allow themethod to be implemented, even if that is the sole purpose of the seconddevice (i.e., amplification may not be required in the subject'sipsilateral ear).

This method has been demonstrated by the present inventor throughexperimentation to significantly improve speech intelligibility foreight BCI users. Cochlear implant users typically suffer significantdeclines in speech understanding in the presence of background noise,and often show the largest declines in single-talker backgrounds, apattern that is also often observed in listeners with hearingimpairment. Users of bilateral hearing aids should also benefit, as wellas users of unilateral devices (hearing aid or cochlear implant),provided that a contralateral microphone and suitable data link areavailable. In addition, the disclosed concept should provide significantbenefit to users of bilateral hearing aids, even if they already showsgood localization abilities and SRM. This is because the techniquedescribed herein improves the SNR, which will be beneficial with orwithout spatial abilities.

Furthermore, there are many listeners whose hearing thresholds are goodenough that aids do not provide benefit, but who still show significantperformance declines in multiple-talker environments. These declines maybe due to presbicusis, or cognitive or attentional deficits. Thedescribed method may provide significant benefit in many of these casesvia the improved SNR described earlier. In these cases, benefit may beobtained by utilizing the described method with two hearing aids.

In the claims, any reference signs placed between parentheses shall notbe construed as limiting the claim, The word “comprising” or “including”does not exclude the presence of elements or steps other than thoselisted in a claim. In a device claim enumerating several means, severalof these means may be embodied by one and the same item of hardware. Theword “a” or “an” preceding an element does not exclude the presence of aplurality of such elements. In any device claim enumerating severalmeans, several of these means may be embodied by one and the same itemof hardware. The mere fact that certain elements are recited in mutuallydifferent dependent claims does not indicate that these elements cannotbe used in combination.

Although the invention has been described in detail for the purpose ofillustration based on what is currently considered to be the mostpractical and preferred embodiments, it is to be understood that suchdetail is solely for that purpose and that the invention is not limitedto the disclosed embodiments, but, on the contrary, is intended to covermodifications and equivalent arrangements that are within the spirit andscope of the appended claims. For example, it is to be understood thatthe present invention contemplates that, to the extent possible, one ormore features of any embodiment can be combined with one or morefeatures of any other embodiment.

What is claimed is:
 1. A method of enhancing binaural representation fora subject, comprising: receiving a first signal and a second signalgenerated by one or more microphones in response to a plurality of soundsources; generating a number of estimated interaural time differencesusing the first signal and the second signal; converting each of thenumber of estimated interaural time differences to a correspondinginteraural level difference; using one or more of the correspondinginteraural level differences to generate an adjusted first signal; andusing the adjusted first signal in a hearing enhancement device togenerate a number of signals delivered to the subject by the hearingenhancement device for enhancing the hearing of the subject.
 2. Themethod according to claim 1, wherein the using one or more of thecorresponding interaural level differences to generate an adjusted firstsignal comprises using each of the corresponding interaural leveldifferences to generate the adjusted first signal.
 3. The methodaccording to claim 2, wherein the using the adjusted first signal in ahearing enhancement device to generate a number of signals delivered tothe subject by the hearing enhancement device for enhancing the hearingof the subject comprises using the adjusted first signal and the secondsignal in the hearing enhancement device to generate the number ofsignals delivered to the subject by the hearing enhancement device forenhancing the hearing of the subject.
 4. The method according to claim1, wherein the hearing enhancement device is a cochlear implant or ahearing aid.
 5. The method according to claim 1, wherein the generatinga number of estimated interaural time differences using the first signaland the second signal comprises: (i) filtering the first signal into afirst plurality of frequency bands and filtering the second signal intoa second plurality of second frequency bands, and (ii) generating thenumber of estimated interaural time differences using a correlationalgorithm and the first plurality of frequency bands and the secondplurality of frequency bands.
 6. The method according to claim 5,wherein the correlation algorithm is a windowed sliding crosscorrelation algorithm.
 7. The method according to claim 5, wherein eachof the estimated interaural time differences and each of thecorresponding interaural level differences is associated with arespective one of the first plurality of frequency bands, and whereinthe using the one or more of the corresponding interaural leveldifferences to generate the adjusted first signal comprises attenuatinga value of each of the first plurality of frequency bands using theassociated corresponding interaural level difference to create aplurality of adjusted frequency bands and summing at least the pluralityof adjusted frequency bands to form the adjusted first signal.
 8. Themethod according to claim 7, wherein the first plurality of frequencybands have a frequency of 1500 Hz or less.
 9. The method according toclaim 1, wherein the converting each of the number of estimatedinteraural time differences to a corresponding interaural leveldifference employs a predetermined scheme.
 10. The method according toclaim 9, wherein the predetermined scheme comprises a look-up table. 11.A method of enhancing hearing of a subject, comprising: receiving afirst signal and a second signal generated by one or more microphones inresponse to a plurality of sound sources; generating a number of firstbinaural cues indicating a spatial position of one or more of the soundsources using the first signal and the second signal; using the firstbinaural cues to create a number of second binaural cues; and using thenumber of second binaural cues and the first signal and the secondsignal in a hearing enhancement device to generate a number of signalsdelivered to the subject by the hearing enhancement device for enhancingthe hearing of the subject.
 12. A method according to claim 11, whereinthe number of first binaural cues comprises a number of estimatedinteraural time differences and the number of second binaural cuescomprises a number of interaural level differences, wherein eachinteraural level difference corresponds to a respective one of theestimated interaural time differences.
 13. A method of enhancingbinaural representation for a subject, comprising: receiving a firstsignal and a second signal generated by one or more microphones inresponse to a plurality of sound sources; generating a number ofestimated binaural cues of a first type using the first signal and thesecond signal; converting each of the number of estimated binaural cuesof the first type to a corresponding binaural cues of a second typedifferent than the first type; using one or more of the correspondingbinaural cues of the second type to generate an adjusted first signal;and using the adjusted first signal in a hearing enhancement device togenerate a number of signals delivered to the subject by a hearingenhancement device for enhancing the hearing of the subject.
 14. Themethod according to claim 13, wherein the using one or more of thecorresponding binaural cues of the second type to generate an adjustedfirst signal comprises using each of the corresponding binaural cues ofthe second type to generate the adjusted first signal.
 15. The methodaccording to claim 14, wherein the using the adjusted first signal in ahearing enhancement device to generate a number of signals delivered tothe subject by the hearing enhancement device for enhancing the hearingof the subject comprises using the adjusted first signal and the secondsignal in the hearing enhancement device to generate the number ofsignals delivered to the subject by the hearing enhancement device forenhancing the hearing of the subject.
 16. The method according to claim13, wherein the hearing enhancement device is a cochlear implant or ahearing aid.
 17. The method according to claim 13, wherein thegenerating a number of estimated binaural cues of the first type usingthe first signal and the second signal comprises: (i) filtering thefirst signal into a first plurality of frequency bands and filtering thesecond signal into a second plurality of second frequency bands, and(ii) generating the number of estimated binaural cues of the first typeusing a correlation algorithm and the first plurality of frequency bandsand the second plurality of frequency bands.
 18. The method according toclaim 17, wherein each of the estimated binaural cues of the first typeand each of the corresponding binaural cues of the second type isassociated with a respective one of the first plurality of frequencybands, and wherein the using the one or more of the correspondingbinaural cues of the second type to generate the adjusted first signalcomprises attenuating a value of each of the first plurality offrequency bands using the associated corresponding binaural cue of thesecond type to create a plurality of adjusted frequency bands andsumming at least the plurality of adjusted frequency bands to form theadjusted first signal.